Calls grow to curb obscene content generated by AI in China

BEIJING – A recent case and the exposure of irregularities in the artificial intelligence sector have reignited public concern over the use of AI tools to create and disseminate obscene content, prompting renewed debate among experts over the ethical and legal boundaries of the technology.

On April 11, the cybersecurity bureau of China’s Ministry of Public Security said the police in Jinhua, Zhejiang province, had solved a case involving the use of AI to produce and distribute vulgar and pornographic content, which led to the detention of eight suspects.

The investigation began after a long-time fan of Chinese comics reported that several accounts in an online group were offering “customised services” – posting obscene images of popular anime characters.

Following the leads, the police dismantled a gang led by a man surnamed Chen and uncovered multiple AI models and databases used to generate obscene content. Investigators allege Chen and his associates profited by using AI tools to create vulgar material featuring comic characters based on client requests.

On April 12, China Central Television (CCTV) reported that some individuals, under the guise of promoting technology, were selling online AI tutorials that teach users how to produce pornographic content. The sellers claim users can generate obscene videos or images within seconds and evade detection by following their instructions.

Some also sell prompt keywords related to clothing, hairstyles and poses for less than 10 yuan (S$1.86) to make the generated content more precise.

CCTV also reported that various AI software programs that profit from generating pornographic content based on user payments are widely available.

The exposure of such content has sparked public anger and drawn attention from technical and legal experts, who raised concerns over its impact on minors.

Cybersecurity expert He Yanzhe from the China Electronics Standardization Institute, told CCTV that domestic apps are subject to content review obligations. However, foreign software may bypass such scrutiny and could introduce viruses or other information security risks once downloaded.

Recently, a Chinese app based on a third-party overseas AI model came under scrutiny for facilitating sexually explicit conversations.

In September, a Shanghai court sentenced the app’s developer and operator to four years and 18 months in prison, respectively, for profiting from the production of obscene and pornographic content. The case is under appeal.

Associate Professor Wang Xiaohua from the East China University of Political Science and Law, highlighted the importance of implementing the interim measures for the management of generative AI services introduced by the Cyberspace Administration of China in 2023.

He told Wenhui Daily that the measures call for stronger oversight of AI developers and operators. He said AI companionship apps should regulate prompt settings during development to prevent the generation of pornographic content and ensure such material is monitored during operations.

Associate Professor Zhu Wei from the China University of Political Science and Law, also cited the interim measures, emphasising that the design and application of large language models must comply with laws and regulations and must not violate ethical norms, public order or public security.

“Under the current legal framework, when pornography in private chats is amplified through an uncontrolled, profit-driven platform, it becomes a social harm that requires technical and service providers to ensure safety,” Prof Zhu said.

Prof Wang noted that while emotional needs and expression are normal for teenagers, AI companionship software should strengthen safeguards for young users. While strictly implementing real-name registration systems, he urged platforms to require minors to provide guardian contact information when registering or downloading apps.

Although much AI-generated content may not meet the legal threshold for obscenity under China’s Criminal Law, lawyer Lei Jiamao said sexually suggestive and provocative content can still seriously harm minors’ physical and mental health, calling for stronger administrative management, supervision and penalties.

The principle of “the best interests of minors” must be upheld by applying the strictest standards to protect young people, the lawyer said.

“It is essential for generative AI service providers to register with the cybersecurity authorities, and legal oversight across the sector should be strengthened,” Prof Zhu added. CHINA DAILY/ASIA NEWS NETWORK